{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab Exercise: Implement batch normalization and dropout\n",
    "\n",
    "In this notebook you will modify your implementation of a convolutional neural network to include batch normalization and dropout of the dense layer. Then, you will validate the changes in your local environment, and finally train and deploy the model using Cloud ML Engine. The objective of this lab is to significantly improve the accuracy of your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "PROJECT = 'my-project-id' # REPLACE WITH YOUR PROJECT ID\n",
    "BUCKET = 'my-bucket-name' # REPLACE WITH YOUR BUCKET NAME\n",
    "REGION = 'us-central1' # REPLACE WITH YOUR BUCKET REGION e.g. us-central1\n",
    "MODEL_TYPE='cnn_batch_norm'  #  'cnn_batch_norm' or 'dnn' or 'cnn'\n",
    "\n",
    "# do not change these\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['REGION'] = REGION\n",
    "os.environ['MODEL_TYPE'] = MODEL_TYPE\n",
    "os.environ['TFVERSION'] = '1.8'  # Tensorflow version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%bash\n",
    "gcloud config set project $PROJECT\n",
    "gcloud config set compute/region $REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO: Add batch normalization to the last dense layer of your CNN**\n",
    "\n",
    "Recall that to implement batch normalization using <code>tf.layers</code> API you need to ensure that the activation function is used after batch normalization is applied to your dense layer. Here's an example of batch normalization of the last dense layer (h3) of the convolutional neural network used in this lab:\n",
    "\n",
    "<pre>\n",
    "  h3 = tf.layers.dense(p2flat, 300, activation=None) #Activation to be added separately after batch normalization\n",
    "\n",
    "  h3 = tf.layers.batch_normalization(h3, \n",
    "             training=(mode == tf.estimator.ModeKeys.TRAIN))\n",
    "             \n",
    "  h3_batch_normed = tf.nn.relu(h3)\n",
    "</pre>\n",
    "\n",
    "Also, note that <code>mode == tf.estimator.ModeKeys.TRAIN</code> evaluates to <code>True</code> only during model training, so that batch norming is bypassed when the model accuracy is evaluated or used for predictions.\n",
    "\n",
    "Open <a href=\"fashionmodel/trainer\">fashionmodel/trainer</a> to find the `model.py` file. Add batch normalization as part of your <code>cnn_batch_norm_model</code> method implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO: Add dropout to the batch normed layer of your CNN**\n",
    "\n",
    "Dropout is another feature of the <code>tf.layers</code> library. As with batch normalization, dropout will only be applied during model training. Dropout can be used on a batch normed layer as in the code snippet below:\n",
    "\n",
    "<pre>\n",
    "  h3_with_dropout = tf.layers.dropout(h3_batch_normed,rate=0.1, \n",
    "                                         training=(mode == tf.estimator.ModeKeys.TRAIN))\n",
    "                    \n",
    "  ylogits = tf.layers.dense(h3_with_dropout, NCLASSES, activation=None)\n",
    "\n",
    "</pre>\n",
    "\n",
    "Open <a href=\"fashionmodel/trainer\">fashionmodel/trainer</a> to find the `model.py` file. Add dropout to the batched normed layer your implemented in the previous cell in the `cnn_batch_norm_model` method.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run as a Python module\n",
    "\n",
    "You should expect that training will take a bit longer than with the previous lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%bash\n",
    "rm -rf fashionmodel.tar.gz fashion_trained\n",
    "gcloud ml-engine local train \\\n",
    "   --module-name=trainer.task \\\n",
    "   --package-path=${PWD}/fashionmodel/trainer \\\n",
    "   -- \\\n",
    "   --output_dir=${PWD}/fashion_trained \\\n",
    "   --train_steps=1 \\\n",
    "   --learning_rate=0.01 \\\n",
    "   --model=$MODEL_TYPE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Make sure that local training completed successfully before training using Cloud ML Engine**\n",
    "\n",
    "Next, go ahead and try training with Cloud ML Engine. Compare the accuracy of the model that uses batch norming and dropout to the accuracy of convolutional neural network you  trained in the previous lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%bash\n",
    "OUTDIR=gs://${BUCKET}/fashion/trained_${MODEL_TYPE}\n",
    "JOBNAME=fashion_${MODEL_TYPE}_$(date -u +%y%m%d_%H%M%S)\n",
    "echo $OUTDIR $REGION $JOBNAME\n",
    "gcloud storage rm --recursive --continue-on-error $OUTDIR\n",
    "gcloud ml-engine jobs submit training $JOBNAME \\\n",
    "   --region=$REGION \\\n",
    "   --module-name=trainer.task \\\n",
    "   --package-path=${PWD}/fashionmodel/trainer \\\n",
    "   --job-dir=$OUTDIR \\\n",
    "   --staging-bucket=gs://$BUCKET \\\n",
    "   --scale-tier=BASIC_GPU \\\n",
    "   --runtime-version=$TFVERSION \\\n",
    "   -- \\\n",
    "   --output_dir=$OUTDIR \\\n",
    "   --train_steps=10000 --learning_rate=0.01 --train_batch_size=512 \\\n",
    "   --model=$MODEL_TYPE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitoring training with TensorBoard\n",
    "\n",
    "Use this cell to launch tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.datalab.ml import TensorBoard\n",
    "TensorBoard().start('gs://{}/fashion/trained_{}'.format(BUCKET, MODEL_TYPE))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for pid in TensorBoard.list()['pid']:\n",
    "  TensorBoard().stop(pid)\n",
    "  print 'Stopped TensorBoard with pid {}'.format(pid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploying and predicting with model\n",
    "\n",
    "**Make sure that the previous traiining step is finished and deploy the model:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%bash\n",
    "MODEL_NAME=\"fashion\"\n",
    "MODEL_VERSION=${MODEL_TYPE}\n",
    "MODEL_LOCATION=$(gcloud storage ls gs://${BUCKET}/fashion/trained_${MODEL_TYPE}/export/exporter | tail -1)\n",
    "echo \"Deleting and deploying $MODEL_NAME $MODEL_VERSION from $MODEL_LOCATION ... this will take a few minutes\"\n",
    "#gcloud ml-engine versions delete ${MODEL_VERSION} --model ${MODEL_NAME}\n",
    "#gcloud ml-engine models delete ${MODEL_NAME}\n",
    "#gcloud ml-engine models create ${MODEL_NAME} --regions $REGION\n",
    "gcloud ml-engine versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version=$TFVERSION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous step of deploying the model can take a few minutes. If it is successful, you should see an output similar to this one:\n",
    "\n",
    "<pre>\n",
    "Created ml engine model [projects/qwiklabs-gcp-27eb45524d98e9a5/models/fashion].\n",
    "Creating version (this might take a few minutes)......\n",
    "...................................................................................................................done.\n",
    "</pre>\n",
    "\n",
    "Next, download a local copy of the Fashion MNIST dataset to use with Cloud ML Engine for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.fashion_mnist.load_data()\n",
    "LABELS = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To predict with the model, save one of the test images as a JavaScript Object Notation (JSON) file. Also, take a look at it as a graphic and notice the expected class value in the title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HEIGHT=28\n",
    "WIDTH=28\n",
    "\n",
    "IMGNO=5500 #CHANGE THIS to get different images\n",
    "\n",
    "#Convert raw image data to a test.json file and persist it to disk\n",
    "import json, codecs\n",
    "jsondata = {'image': test_images[IMGNO].reshape(HEIGHT, WIDTH).tolist()}\n",
    "json.dump(jsondata, codecs.open('test.json', 'w', encoding='utf-8'))\n",
    "\n",
    "#Take a look at a sample image and the correct label from the test dataset\n",
    "import matplotlib.pyplot as plt\n",
    "plt.imshow(test_images[IMGNO].reshape(HEIGHT, WIDTH))\n",
    "title = plt.title('{} / Class #{}'.format(LABELS[test_labels[IMGNO]], test_labels[IMGNO]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send the file to the prediction service and check whether the model you trained returns the correct prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%bash\n",
    "gcloud ml-engine predict \\\n",
    "   --model=fashion \\\n",
    "   --version=${MODEL_TYPE} \\\n",
    "   --json-instances=./test.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "# Copyright 2017 Google Inc. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#      http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
